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Lecture Outlines 



o Review on probability theory 

■ Conditional probability and Bayes' rule 

■ Random variables and the distribution functions 

■ Multiple random variables 

■ The concept of expectation and higher order statistics 
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Conditional probability 



o Probability of event A given that event B has occurred 

p [A \B] = P[AnB] 
1 P[B] 

olf A, B are independent: P[AnB] = P[A]P[B] 

■ At knowledge of the occurrence of either event, A or B, tells us no 
more about the probability of occurrence of the other event than we 
knew without that knowledge 

■ Events A and B that satisfy this condition are said to be independent, 
and then P[A|fl]=P>[A] 



Bayes' Rule 



o Posterior probability: inference based on make assumptions 



P[AnB] = P[A\B]P[B] 
P[AnB] = P[B\A]P[A] 




► 


P[A|£]P[£] = P[B|A]P[A] 






f 






1 P[A] 
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Law of Total Probability 

Let Bj B x be N mutually exclusive events^ whose union gives the sample space Q. 
Hence the events B constitute a partition of £2 

Now consider an event A n a subset of Q. This event can be represented as 

A-Aflfl- An(B l UB a U...UB H )-(AnB 1 >U(Af|B a )U (Af)B N ) 




Since the B i are mutually exclusive 

p{a)=p{a n b, )+ p(a n b 2 }+ .„+ p{a n * v ) 

P{A) = P(A I B y )P{B X ) + ... + P(A I B N )P(B N ) = £ | ^ W*t } 

k=i 



* Two events A, B are mutually exclusive if they cannot occur at the same time: A n B = 0 
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General Bayes Rule 



We now pose the following question: Given that the events has occurred. What is 
the probability that any single one of the event B 's occur? 




This is known as the Bayes rule 



Example 



Radar, a remote sensing system, operates by transmitting a sequence of pulses and has its receiver listen to 
echoes produced by a target (e.g., aircraft) that could be present in its surveillance area. 



Let the events A and B be defined as follows: 

A = { a target is present in the area under surveillance } 
A c = {there is no target in the area) 
B = { the radar receiver detects a target ] 

In the radar detection problem, there are three probabilities of particular interest: 

P[A] probability that a target is present in the area; this probability is called the 
prior probability. 

P[B\A] probability that the radar receiver detects a target, given that a target is 
actually present in the area; this second probability is called the probability 
of detection. 

P[i?|A c ] probability that the radar receiver detects a target in the area, given that there 
is no target in the surveillance area; this third probability is called the 

probability of false alarm. 



Given: 



P[A] = 


0.02 


P[B\A] 


= 0.99 


P[B\A Q ] 


= 0.01 



Required: 



P[A\B] 



The probability that a 
target is present in the 
surveillance area given that 
the radar receiver has made 
a target detection. 



Posterior probability likelihood 



P[A\B] 



Provide the language for inference 



solution 

P[g|A]P[A] 



Prior probability 



P[B] 



evidence 



P[A] = 


0.02 


P[£|A] 


= 0.99 


P[fi|A c ] 


= 0.01 



P[fi|A]P[A] + P[£|A C ]P[A C ] 

0.99 x 0.02 
0.99 x 0.02 + 0.01 x0.98 

0.0198 
0.0296 

0.69 



o Real experiments involve using 
one or more real valued 
quantities called random 
variables 

oThe outcomes are associated 
with some numerical values of 
interest (random variable) 



The random variable is a 

function whose domain is a 
sample space and whose range 
is some set of real numbers 



Variables 



Random 
variable 00 



experiment 




— 00 



Probability 

T 1 



\ 



1 0 



Illustration of the relationship between sample 
space, random variables, and probability 



Discrete and Continuous Random Variables 

o Random variables can discrete, e.g., the number of heads in three 
consecutive coin tosses, or continuous, the weight of a class member. 
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Probability/Cumulative Mass Function 



o A probability mass (distribution) function, p(X=x), is a function that tells 
us the probability of x, an observation of X, assuming a specific value 

o The cumulative mass (distribution) function, F x (x), indicates the 
probability of X assuming a value less then or equal tox 

F x (x) = P[X<x] for all x 

■ Lies between zero and one: F x (x=-co)=0 and F x (x=oo)=l 

■ Monotone non-decreasing: F x (xj) < F x (x 2 ) for x 2 < x 2 
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Probability/Cumulative Density Function (cont'd) 



oFor continuos variables: 

■ Probability Mass Function -> Probability Density function (pdf), 

• Non-negativity: f^x)>0 

f x {x)dx — 1 



Cumulative Mass Function -> Cumulative Density Function (CDF) 

• P(xj<X<x 2 )=F x (x 2 )-F x (xj) =Qf x (x)dx 

* ^(^)=/- 7 oo/^)^ 
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Example 1 Uniform Distribution 



x < a 



a 



, a<x<b 
x > b 



0, 

x - a 
b- a 



x<a 



a<x< b 



x > b 



1.0 




0 a 



Example 2 Bernoulli Random Variable 

1 2 

1-/7 X = 0 

p, X = 1 

0, otherwise 



Probability- 
mass 
function 

P\X = x] 



0 1 

Illustrating the probability mass 
function for a fair coin-tossing experiment 



Multiple Random Variables 



oThe joint distribution function F X Y (x,y) is the probability that the random 
variable X is less than or equal to a specified value x, and that the 
random variable Y is less than or equal to another specified value y 

F XY (x,y) = P[X<x,Y<y] = f f f XY (x,y)dxdy 



—00 —00 



oThe joint probability density function f X j[x>y) 

2 

d F x Y {x,y) 

For continuous R.V. fy y(jC ? = — 

dxdy 

00 00 

Total volume under the joint f ( X , y) &X dv = 1 

probability density graph J J ^> ^ 



—00 " —oo 15 



The joint probability density function 

oThe joint probability density function f Xt) y( x ->y) contains all the 
conceivable information on the two continuous random variables X 
and Y that is needed for the probability analysis of joint random 
variables 
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Marginal probability 



x „ 00 



F X (X) = F XY (X 9 <X))=I j f XY (x,y)dxdy 



—00 —00 

Marginal probability distribution 



r 00 r y 

F y(y) = F X,Y(^y)= J J /*, A y) & dy 



d f- 



—00 —00 



-00 

Marginal probability density 

.00 



—00 
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Conditional Probability Density Function 



Axioms of probability 



oThe conditional probability density function of Y, such that X = x 

fy y( x > y) 

f Y \x(y\x) = f Y (y\x) = Jx f { " J 

f x ( x ) 

oBy cross multiplication: 

f x y (*» y) = fr(y I x )fxM 



Joint 
probability 



conditional Marginal 
probability probability 



fi(y\x)zo 



r 00 

I f Y (y\ x ) d y = 

-00 



= 1 



o Y, X are independent if f X Y (^y) = fxMfyM 
and then f Y (y\x) = f Y (y) 
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Expectation: first order statistical average 



oThe probability mas function of a random variable X provides us with 
several numbers, the probabilities of all the possible values of X 

olt would be desirable to summarize this information in a single 
representative number 

oThis is accomplished by the expectation of X, which is a weighted (in 
proportion to probabilities) average of the possible values of X 

o Expectation of X Locates the center of gravity of the area under the 
probability density curve of the random variable X 



00 




X 

For discrete R.V. 



E[X] = 2> x (*) 



For continuous R.V. 
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Properties of Expectation 



oLinearity: if z = X+Y E[Z] = E[X] + E[r] 



E 



ft 

i=\ 



i=\ 



oStatistical independence: if X and Y are independent R.Vs then: 

E[XY] = E[X]E[F] 

oNotethat: l[g(X)] = \ g(x)f x (x) dx 
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Higher Order Statistics 



Yl i Yl 

o n th moment of the probability distribution of a random varia b\eX E[X ] = x f x {x) dx 



o Mean square value of X E[X ] = J x f x (x) dx 



—00 

co 2 



-00 

00 



o Central moments E[(X-// x ) n ] = f (x-// x f/ x W^ 

J -oo 



o Variance var[X] = l(X-ju x ) = J (x-ju x )f x ( x ) &c = E[X 2 ]-/4 

—00 

Cy 

oChebyshev inequality P[|X-// X | > e] < -| 

6* 
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Moments of Multiple Random Variables 

. i 00 00 , 

o The joint moments (expectations of X* F' for positive i,j) E[X l Y ] = [ [ jt ;y / x y (;c, y) dx dy 

—00 —00 

00 00 

o Correlation (if i=j=l) E[XY] = J | y) (k 

—00 —00 

o Covariance of X and Yis the correlation of the centered random variables (X-E[X\) and (F-E[Y]) 

ju x = E[X] and ju Y = E[Y] 

2 2 
°X an d °Y are variances of X and F 

o Covariance of uncorrelated R.VsX and Y cov[XF] = 0 



cov[XY] = E[(X-E[X])(7-E[7])]= E[XY] - ju x ju y 
o Correlation coefficient of X and Y P(X 9 Y) = ^2XljLfJ 

G X G Y 



Uncorrelated vs. Orthogonal R.Vs 



The two random variables X and Y are said to be uncorrelated if, and only if, their 
covariance is zero; that is, 

cov[XY] = 0 

They are said to be orthogonal if and only if their correlation is zero; that is, 

E[xr] = o 

If one of the random variables X and Y or both have zero means, and if they are 
orthogonal, then they are uncorrelated, and vice versa 



cov[XY] = E[(X-E[X])(7-E[7])]= E[XY] - ju x ju y 
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Questions 
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